Fluoroscopic tracking and visualization system

ABSTRACT

A system employs a tracker and a set of substantially non-shadowing point markers, arranged in a fixed pattern or set in a fluoroscope calibration fixture that is imaged in each shot. The fixture is preferably affixed to the image detector of the fluoroscope, and tracking elements secured with respect to the fixture and at least one of a tool and the patient, provide respective position data irrespective of movement. A marker detection module identifies markers imaged in each shot, and a processor applies the known marker positions to model the projection geometry, e.g., camera axis and focus, for the shot and, together with the tracked tool position, form a corrected tool navigation image. In one embodiment an inverting distortion correction converts the tracked or actual location of the tool and displays the tool on the fluoroscopic image to guide the surgeon in tool navigation. In another aspect of the invention, the fluoroscope takes a series of frames while rotating in a plane about the patient, and the camera models derived from the marker images in each frame are applied to define a common center and coordinate axes in the imaged tissue region to which all of the fluoroscope view may be registered. The processor then filters and back-projects the image data or otherwise forms a volume image data set corresponding to the region of tissue being imaged, and desired fluoro-CT planar images of a the imaged patient volume are constructed from this data set. Planes may then be constructed and displayed without requiring complex tracking and image correlation systems previously needed for operating-room management of MRI, CT or PET study image data. Further, the fluoro-CT images thus constructed may be directly registered to preoperative MRI, CT or PET 3D image data, or may obviate the need for such preoperative imaging. Preferably, the tracker employs electromagnetic tracking elements, as shown for example in U.S. Pat. No. 5,967,980, to generate and/or detect electromagnetic field components unobstructed by the patient and intervening structures, and to determine coordinates directly referenced to the patient, the tool or the camera. The calibration fixture may be implemented with BBs in a radiolucent block of structural foam, and/or may be implemented by microlithographic techniques, in which case magnetic tracking elements may be simultaneously formed in registry with the markers on a sheet that mounts to the camera, is incorporated in a radiographic support table, or otherwise positioned to be imaged in each shot.

BACKGROUND

[0001] The present invention relates to medical and surgical imaging,and in particular to intraoperative or perioperative imaging in whichimages are formed of a region of the patient's body and a surgical toolor instrument is applied thereto, and the images aid in an ongoingprocedure. It is of a special utility in surgical procedures such asbrain surgery and arthroscopic procedures on the knee, wrist, shoulderor spine, as well as certain types of angiography, cardiac procedures,interventional radiology and biopsies in which x-ray images may be takento display, correct the position of, or otherwise navigate a tool orinstrument involved in the procedure.

[0002] Several areas of surgery have required very precise planning andcontrol for the placement of an elongated probe or other article intissue or bone that is internal or difficult to view directly. Inparticular, for brain surgery, stereotactic frames to define the entrypoint, probe angle and probe depth are used to access a site in thebrain, generally in conjunction with previously compiledthree-dimensional diagnostic images such as MRI, PET or CT scan imageswhich provide accurate tissue images. For placement of pedicle screws inthe spine, where visual and fluoroscopic imaging directions cannotcapture an axial view necessary to center the profile of an insertionpath in bone, such systems have also been useful.

[0003] When used with existing CT, PET or MRI image sets, thesepreviously recorded diagnostic image sets themselves define a threedimensional rectilinear coordinate system, by virtue of their precisionscan formation or the spatial mathematics of their reconstructionalgorithms. However, it may be necessary to correlate the availablefluoroscopic views and anatomical features visible from the surface orin fluoroscopic images with features in the 3-D diagnostic images andwith the external coordinates of the tools being employed. This is oftendone by providing implanted fiducials, and adding externally visible ortrackable markers that may be imaged, and using a keyboard or mouse toidentify fiducials in the various images, and thus identify common setsof coordinate registration points in the different images, that may alsobe trackable in an automated way by an external coordinate measurementdevice, such as a suitably programmed off-the-shelf optical trackingassembly. Instead of imageable fiducials, which may for example beimaged in both fluoroscopic and MRI or CT images, such systems can alsooperate to a large extent with simple optical tracking of the surgicaltool, and may employ an initialization protocol wherein the surgeontouches or points at a number of bony prominences or other recognizableanatomic features in order to define the external coordinates inrelation to the patient anatomy and to initiate software tracking ofthose features.

[0004] Generally, systems of this type operate with an image displaywhich is positioned in the surgeon's field of view, and which displays afew panels such as a selected MRI image and several x-ray orfluoroscopic views taken from different angles. The three-dimensionaldiagnostic images typically have a spatial resolution that is bothrectilinear and accurate to within a very small tolerance, e.g., towithin one millimeter or less. The fluoroscopic views by contrast aredistorted, and they are shadowgraphic in that they represent the densityof all tissue through which the conical x-ray beam has passed. In toolnavigation systems of this type, the display visible to the surgeon mayshow an image of the surgical tool, biopsy instrument, pedicle screw,probe or the like projected onto a fluoroscopic image, so that thesurgeon may visualize the orientation of the surgical instrument inrelation to the imaged patient anatomy, while an appropriatereconstructed CT or MRI image, which may correspond to the trackedcoordinates of the probe tip, is also displayed.

[0005] Among the systems which have been proposed for effecting suchdisplays, many rely on closely tracking the position and orientation ofthe surgical instrument in external coordinates. The various sets ofcoordinates may be defined by robotic mechanical links and encoders, ormore usually, are defined by a fixed patient support, two or morereceivers such as video cameras which may be fixed to the support, and aplurality of signaling elements attached to a guide or frame on thesurgical instrument that enable the position and orientation of the toolwith respect to the patient support and camera frame to be automaticallydetermined by triangulation, so that various transformations betweenrespective coordinates may be computed. Three-dimensional trackingsystems employing two video cameras and a plurality of emitters or otherposition signaling elements have long been commercially available andare readily adapted to such operating room systems. Similar systems mayalso determine external position coordinates using commerciallyavailable acoustic ranging systems in which three or more acousticemitters are actuated and their sounds detected at plural receivers todetermine their relative distances from the detecting assemblies, andthus define by simple triangulation the position and orientation of theframes or supports on which the emitters are mounted. When trackedfiducials appear in the diagnostic images, it is possible to define atransformation between operating room coordinates and the coordinates ofthe image.

[0006] In general, the feasibility or utility of a system of this typedepends on a number of factors such as cost, accuracy, dependability,ease of use, speed of operation and the like. Intraoperative x-rayimages taken by C-arm fluoroscopes alone have both a high degree ofdistortion and a low degree of repeatability, due largely todeformations of the basic source and camera assembly, and to intrinsicvariability of positioning and image distortion properties of thecamera. In an intraoperative sterile field, such devices must be draped,which may impair optical or acoustic signal paths of the signal elementsthey employ to track the patient, tool or camera

[0007] More recently, a number of systems have been proposed in whichthe accuracy of the 3-D diagnostic data image sets is exploited toenhance accuracy of operating room images, by matching these 3-D imagesto patterns appearing in intraoperative fluoroscope images. Thesesystems may require tracking and matching edge profiles of bones,morphologically deforming one image onto another to determine acoordinate transform, or other correlation process. The procedure ofcorrelating the lesser quality and non-planar fluoroscopic images withplanes in the 3-D image data sets may be time-consuming, and in thosetechniques that rely on fiducials or added markers, the processingnecessary to identify and correlate markers between various sets ofimages may require the surgeon to follow a lengthy initializationprotocol, or may be a slow and computationally intensive procedure. Allof these factors have affected the speed and utility of intraoperativeimage guidance or navigation systems.

[0008] Correlation of patient anatomy or intraoperative fluoroscopicimages with precompiled 3-D diagnostic image data sets may also becomplicated by intervening movement of the imaged structures,particularly soft tissue structures, between the times of originalimaging and the intraoperative procedure. Thus, transformations betweenthree or more coordinate systems for two sets of images and the physicalcoordinates in the operating room may require a large number ofregistration points to provide an effective correlation. For spinaltracking to position pedicle screws it may be necessary to initializethe tracking assembly on ten or more points on a single vertebra toachieve suitable accuracy. In cases where a growing tumor or evolvingcondition actually changes the tissue dimension or position betweenimaging sessions, further confounding factors may appear.

[0009] When the purpose of image guided tracking is to define anoperation on a rigid or bony structure near the surface, as is the casein placing pedicle screws in the spine, the registration mayalternatively be effected without ongoing reference to tracking images,by using a computer modeling procedure in which a tool tip is touched toand initialized on each of several bony prominences to establish theircoordinates and disposition, after which movement of the spine as awhole is modeled by optically initially registering and then trackingthe tool in relation to the position of those prominences, whilemechanically modeling a virtual representation of the spine with atracking element or frame attached to the spine. Such a proceduredispenses with the time-consuming and computationally intensivecorrelation of different image sets from different sources, and, bysubstituting optical tracking of points, may eliminate or reduce thenumber of x-ray exposures required to effectively determine the toolposition in relation to the patient anatomy with the required degree ofprecision.

[0010] However, each of the foregoing approaches, correlating highquality image data sets with more distorted shadowgraphic projectionimages and using tracking data to show tool position, or fixing a finiteset of points on a dynamic anatomical model on which extrinsicallydetected tool coordinates are superimposed, results in a process wherebymachine calculations produce either a synthetic image or select anexisting data base diagnostic plane to guide the surgeon in relation tocurrent tool position. While various jigs and proprietary subassemblieshave been devised to make each individual coordinate sensing or imagehandling system easier to use or reasonably reliable, the field remainsunnecessarily complex. Not only do systems often require correlation ofdiverse sets of images and extensive point-by-point initialization ofthe operating, tracking and image space coordinates or features, butthey are subject to constraints due to the proprietary restrictions ofdiverse hardware manufacturers, the physical limitations imposed bytracking systems and the complex programming task of interfacing withmany different image sources in addition to determining their scale,orientation, and relationship to other images and coordinates of thesystem.

[0011] Several proposals have been made that fluoroscope images becorrected to enhance their accuracy. This is a complex undertaking,since the nature of the fluoroscope's 3D to 2D projective imagingresults in loss of a great deal of information in each shot, so thereverse transformation is highly underdetermined. Changes in imagingparameters due to camera and source position and orientation that occurwith each shot further complicate the problem. This area has beenaddressed to some extent by one manufacturer which has provided a morerigid and isocentric C-arm structure. The added positional precision ofthat imaging system offers the prospect that, by taking a large set offluoroscopic shots of an inunobilized patient composed under determinedconditions, one may be able to undertake some form of planar imagereconstruction. However, this appears to be computationally veryexpensive, and the current state of the art suggests that while it maybe possible to produce corrected fluoroscopic image data sets withsomewhat less costly equipment than that required for conventional CTimaging, intra-operative fluoroscopic image guidance will continue torequire access to MRI, PET or CT data sets, and to rely on extensivesurgical input and set-up for tracking systems that allow position orimage correlations to be performed.

[0012] Thus, it remains highly desirable to utilize simple, low-dose andlow cost fluoroscope images for surgical guidance, yet also to achieveenhanced accuracy for critical tool positioning.

[0013] It would be desirable to provide an improved image guidednavigation system for a surgical instrument.

[0014] It would also be desirable to provide such an image guided systemwhich operates with a C-arm fluoroscope to produce enhanced images andinformation.

[0015] It would also be desirable to provide an image-guided surgicalnavigation system adaptable to a fluoroscope that accurately depictstool position.

SUMMARY OF THE INVENTION

[0016] One or more of the foregoing features and other desirable endsare achieved in a method or system of the present invention wherein anx-ray imaging machine of movable angulation, such as a fluoroscope, isoperated to form reference or navigation images of a patient undergoinga procedure. A tracking system employs a tracking element affixed toeach of the imaging machine and tool, and preferably to the patient aswell, to provide respective position data for the tool, the fluoroscopeand patient, while a fixed volume array of markers, which is alsotracked, is imaged in each frame. Preferably the array of markers isaffixed to the detector assembly of the imaging machine, where a singletracking element determines position of the fluoroscope and entire arrayof markers. The fluoroscope may itself also provide furthershot-specific indexing or identification data of conventional type, suchas time, settings or the like. A processor then applies the positiondata from the tracking system, and operates on the imaged markers toproduce a correct tool navigation image for surgical guidance.

[0017] The markers are preferably arranged in a known pattern ofsubstantially non-shadowing point elements positioned in differentplanes. These may be rigidly spaced apart in a predefined configurationin an assembly attached to the fluoroscope, so that the physicalposition of each marker is known exactly in a fixed fluoroscope-basedcoordinate system, and the positions may, for example, be stored in atable. A single tracking element may be affixed on the marker assembly,which may in turn be locked in a fixed position on the fluoroscope, sothat the fluoroscope and marker positions are known in relation to thetool and the patient. Alternatively, one or more separate arrays ofmarkers may be independently positioned and each tracked by a separatetracking element.

[0018] In each fluoroscopic image, the processor identifies a subset ofthe markers and recovers geometric camera calibration parameters fromthe imaged marker positions. These calibration parameters then allowaccurate reference between the recorded image and the tool and patientcoordinates measured by the trackers. The processor may also receivepatient identification data of a conventional type to display or recordwith the shot. In one embodiment the processor computes the calibrationas well as geometric distortion due to the imaging process, and convertsthe tracked or actual location of the tool to a distorted tool imageposition at which the display projects a representation of the tool ontothe fluoroscopic image to guide the surgeon in tool navigation.

[0019] In this aspect of the invention, the processor identifies markersin the image, and employs the geometry of the identified markers tomodel the effective source and camera projection geometry each time ashot is taken, e.g., to effectively define its focus and imagingcharacteristics for each frame. These parameters are then used tocompute the projection of the tool in the fluoroscope image.

[0020] In yet a further aspect of the invention, the fluoroscope isoperated to take a series of shots in progressively varying orientationsand positions as the camera and source are moved about the patient.Accurate calibration for multiple images is then employed to allowthree-dimensional reconstruction of the image data. The processorapplies a reconstruction operation or procedure, for example, backprojection of the registered images to form a volume image data set,e.g., a three dimensional set of image density values of a tissuevolume. The initial set of fluoroscopic images may, for example, beacquired by taking a series of views rotating the fluoroscope in a fixedplane about a target region of tissue. A common center and coordinateaxes are determined for the reconstructed volume, such that the volumeimage data set constructed from the images corresponds to the targetregion. Image planes are then directly constructed and displayed fromthis volume image data set.

[0021] The resultant fluoro-CT images are geometrically comparable toconventional diagnostic image sets of the imaged volume, and obviate theneed for complex tracking and image correlation systems otherwiseproposed or required for operating-room management and display ofpre-operatively acquired volumetric data sets with intraoperative fluoroimages. In accordance with a still further aspect of the invention, thisreconstructed fluoro-CT data set is then registered to or transformed tothe image space coordinates of a preoperative PET, MRI or CT data setfor simultaneous display of both sets of images. In other embodiments,the system of the present invention may be used simply for the purposeof intraoperatively registering preoperative 3D image data to thepatient tissue. In accordance with this aspect of the invention, a setof fluoro-CT image data is constructed as described above, and these areregistered to preoperative 3D image data by mutual information, contourmatching or other correlation procedure. This provides a directregistration of the preoperative data to tracking coordinates withoutrequiring the surgeon to place and image fiducials, touch and enterskeletal or surface registration points, or perform invasivepre-operation image registration protocols.

[0022] The tracking elements of the tracking system may comprise variousposition-indicating elements or markers which operate optically,ultrasonically, electromagnetically or otherwise, and the trackingsystem itself may include hybrid software-mediated elements or stepswherein a pointer or tool of defined geometry is tracked as it touchesfiducials or markers in order to enter or initialize positioncoordinates in a tracking system that operates by triangulating paths,angles or distances to various signal emitting or reflecting markers. Ahybrid tracking system may also be used, including one or more roboticelements which physically encode mechanical positions of linkages orsupports as part of one or more of the tracking measurements being made.Preferably, however, the tracking system employs electromagnetictracking elements such as shown in U.S. Pat. No. 5,967,980, to generateand/or detect electromagnetic field components that pass through or aresubstantially unobstructed by the patient and intervening structures,and to directly determine coordinates in three or more dimensionsreferenced to the tool, the patient or the fluoroscope to which theelements are attached.

[0023] A single tracking element may be affixed to each of thefluoroscope, the patient, and the surgical tool. One presently preferredembodiment of a tracking element employs a magnetic field element, suchas one configured with three mutually orthogonal coils, that otherwiseoperates as a substantially point-origin field generator or fieldsensor. The element may have a rigid or oriented housing, so that whenattached to a rigid object, the tracked coordinates of the element yieldall coordinates, with only a defined constant offset, of the objectitself. The element may be energized as a field generator, or sampled asa field sensor, to produce or detect a field modulated in phase,frequency or time so that some or all of the x-, y-, z-, roll-, pitch-,and yaw coordinates of each tracking element, and thus its associatedobject, are quickly and accurately determined. A table of positioncorrection factors or characteristics may be compiled for one or more ofthe tracking elements to correct for the effects of electromagneticshunting or other forms of interference with the generator or receiverwhich may occur when positioned in a region near to the body of thefluoroscope. This allows a magnetic tracking element to be placed quiteclose to the imaging assembly or other conductive structure and achievehigh position tracking accuracy or resolution. In particular, one ormore tracking elements may be mounted directly on the fluoroscope and/oron calibration fixtures positioned close to the image detector of thefluoroscope to define camera and imaging parameters relative to anothertracker which may move with the patient or with a tool. Variousalternative magnetic generating and sensing assemblies may be used forthe tracking component, such as ones having a tetrahedrally-disposedgenerating element and a single sensing/receiving coil, or ones having amultipole generating assembly that defines a suitably detectable spatialfield.

BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The invention will be understood from the description and claimsherein, viewed in light of the prior art, and taken together with theFigures illustrating several basic embodiments and representativedetails of construction, wherein

[0025]FIG. 1 illustrates a fluoroscopic image and tool navigation systemin accordance with one embodiment of the present invention;

[0026]FIG. 1A illustrates camera imaging of a tissue region with thesystem of FIG. 1;

[0027]FIG. 2 illustrate representative navigation images of oneembodiment of the system of FIG. 1;

[0028]FIG. 2A illustrates the display of fluoroscope orientation in apreferred, implementation of the system of FIG. 1;

[0029]FIG. 3 shows details of one camera calibration sub-assembly usefulin the embodiment of FIG. 1;

[0030]FIG. 3A shows another calibration, sub-assembly of the invention;

[0031]FIG. 4 is a flow chart showing image processing and tool trackingin accordance with a first aspect of the invention;

[0032]FIG. 5 illustrates operation of a second aspect of the invention;

[0033]FIG. 6 illustrates a sheet fixture for use with the invention andhaving combined calibration and tracking elements;

[0034]FIG. 7 illustrates camera calibrations corresponding to thefluoroscope poses illustrated in FIG. 1A and used for the operationillustrated in FIG. 5; and

[0035]FIG. 8 illustrates operation of the system to registerpreoperative images to a patient.

DETAILED DESCRIPTION OF THE INVENTION

[0036]FIG. 1 illustrates elements of a basic embodiment of a system 10in accordance with the present invention for use in an operating roomenvironment. As shown, the system 10 includes a fluoroscope 20, a workstation 30 having one or more displays 32 and a keyboard/mouse or otheruser interface 34, and a plurality of tracking elements T1, T2, T3. Thefluoroscope 20 is illustrated as a C-arm fluoroscope in which an x-raysource 22 is mounted on a structural member or C-arm 26 opposite to anx-ray receiving and detecting unit, referred to herein as an imagingassembly 24. The C-arm moves about a patient for producing twodimensional projection images of the patient from different angles Thepatient remains positioned between the source and the camera, and may,for example, be situated on a table or other support, although thepatient may move. The tracking elements, described further below, aremounted such that one element T1 is affixed to, incorporated in orotherwise secured against movement with respect to a surgical tool orprobe 40. A second tracking unit T2 is fixed on or in relation to thefluoroscope 20, and a third tracking unit T3 fixed on or in relation tothe patient. The surgical tool may be a rigid probe as shown in FIG. 1,allowing the tracker T1 to be fixed at any known or convenient position,such as on its handle, or the tool may be a flexible tool, such as acatheter, flexible endoscope or an articulated tool. In the lattercases, the tracker T1 is preferably a small, localized elementpositioned in or at the operative tip of the tool as shown by cathetertracker T1′ in FIG. 1, to track coordinates of the tip within the bodyof the patient.

[0037] As will be understood by those skilled in the art, fluoroscopestypically operate with an x-ray source 22 positioned opposite the cameraor image sensing assembly 24. While in some systems, the X-ray source isfixed overhead, and the camera is located below a patient support, thediscussion below will be illustrated with regard to the more complexcase of a typical C-arm fluoroscope, in which the source and camera areconnected by a structural member, the C-arm, that allows movement of thesource and camera assembly about the patient so it may be positioned toproduce x-ray views from different angles or perspectives. In thesedevices, the imaging beam generally diverges at an angle, the relativelocations and orientations of the source and camera vary with positiondue to structural flexing and mechanical looseness, and the position ofboth the source and the camera with respect to the patient and/or a toolwhich it is desired to track may also vary in different shots.

[0038] The imaging beam illustrated by B in FIG. 1 diverges from thesource 22 in a generally truncated conical beam shape, and the C-arm 26is movable along a generally arcuate path to position the source andcamera for imaging from different directions. This generally involvespositioning the camera assembly 24 as close as possible behind therelevant tissue or operating area of the patient, while the C-armassembly is moved roughly about a targeted imaging center to the desiredviewing angle. The C-arm or other beam structure 26 may be a somewhatflexible structure, subject to bending, deflection or sagging as thesource and camera move to different positions around the patient, andthe C-arm may also have other forms of dimensional variation orlooseness, such as drive gear backlash, compressible elastomericsuspension components or the like, which may contribute to variationsand non-repeatability of the relative disposition and alignment of thesource and camera with respect to each other, and with respect to thepatient, as the assembly is moved to different positions. The C-arm mayalso move eccentrically or translationally to allow better clearance ofthe patient support table. The bending deflections of the C-arm assemblymay vary the actual position of the source 22 by almost a centimeter ormore with respect to the image detector, and displace it from a nominalposition which may be indicated, for example, by an encoder present inthe fluoroscope stand or C-arm positioning assembly. These variationsmay therefore be significant.

[0039]FIG. 1A illustrates the fluoroscope 20 in two different imagingpositions, with a first position shown in solid line, and a secondposition in dashed line phantom. In the first position, a tissue volumeV is imaged with a divergent beam from the above right, and a virtualbeam origin or focal point at F, while the image from the secondposition catches a largely overlapping but partly distinct tissue volumewith a divergent beam from the upper left, and a different focal pointF′. The distances from points F, F′ to the camera may be different, andthe camera itself may shift and tilt with respect to the beam and itscenter axis, respectively. In practice, the x-ray beam is generallyaimed by its center ray, whose intersection with the imaging plane,referred to as the piercing point, may be visually estimated by aimingthe assembly with a laser pointing beam affixed to the source. The x-raybeam may be considered to have a virtual origin or focal point F at theapex of the cone beam. Generally, the camera assembly 24 is positionedclose to the patient, but is subject to constraints posed by theoperating table, the nature of the surgical approach, and the necessarytools, staging, clamps and the like, so that imaging of a tissue volumesomewhat off the beam center line, and at different distances along thebeam, may occur. As noted above, flexing of the C-arm also changes thedistance to the focal point F and this also may slightly vary theangular disposition of the beam to the camera, so this shifting geometrymay affect the fluoroscope images.

[0040] Furthermore, the camera 24 may utilize an image sensing unit thatitself introduces further distortions into the received distribution ofimage radiation. For example, the unit may involve a detector thatemploys a phosphor surface of generally curved contour to convert thex-ray image intensity distribution to a free electron distribution. Sucha curved phosphor screen is generally placed over an electron multiplieror image intensifier assembly that provides an enhanced output videosignal, but may further introduce a form of electron optical distortionthat depends upon the intensifier geometry and varies with theorientation of the camera assembly in the earth's magnetic field. Otherconfigurations of image detectors are also known or proposed, such asdigital x-ray detectors or flat semiconductor arrays, which may havedifferent imaging-end fidelity characteristics. In any case, deflectionor physical movement of the camera itself as well as electron/opticaldistortion from the camera geometry, image detector and variations dueto gravitational, magnetic or electromagnetic fields can all enter theimage reception and affect the projective geometry and other distortionof the final image produced by the assembly.

[0041] The foregoing aspects of imaging system variability are addressedby the present invention by using tracking elements in conjunction witha camera calibration fixture or correction assembly to providefluoroscopic images of enhanced accuracy for tool navigation andworkstation display.

[0042] A more detailed description of the operation of the presentinvention follows, and proceeds initially from 1) a mechanism foreffectively characterizing camera imaging parameters while addressingdistortion in each image frame or shot of a C-arm fluoroscope to 2)using these parameters to reconstructing a 3D volume that is dynamicallyreferenced to the patient and the tool; and finally 3) fusing thedynamically referenced 3D volume with preoperative volumetric data. Theequipment and procedure has two components, a first component providedby a tracking assembly which determines position of a fluoroscopecalibration fixture relative to one or both of the tool and patientbody, and a second component provided by a processor operating on eachimage that characterizes or models the geometry of the camera andperforms all subsequent processing. This is done by providing acalibration fixture that contains an array of markers, which is eithertracked as a rigid unit or affixed to the camera, while the imagedposition of the markers in each fluoroscope shot serves to characterizethe imaging geometry so as to allow correction of imaged features atmeasured distances from the camera, and permit registration ofsuccessive images in different poses.

[0043] In accordance with a principal aspect of the present invention,when tracked relative to a tool, the surgical instrument display 40′ ofFIGS. 1 and 2 is effected by determining tool position, focus andimaging axis, and rendering the instrument in conjunction with one ormore of the three types of images mentioned above. In one embodiment,the processor determines an image distortion inverse transform andprojects a distorted or transformed tool graphic or image on thefluoroscopic view. In another aspect, the processor determines thecamera geometry for each image and transforms the set of fluoroscopicimages such that the screen coordinates of display 33 are similar oraligned with the operating coordinates as measured by tracking elementsT2, T3. This calibration results in more accurate tool tracking andrepresentation over time. As further discussed in regard to FIG. 5below, the image data of an imaging sequence for a region of tissueabout a common origin may be back-projected or otherwise processed todefine a three dimensional stack of fluoro-CT images. The invention thusallows a relatively inexpensive C-arm fluoroscope to achieve accuracyand registration to prepare CT images for tool guidance andreconstruction of arbitrary planes in the imaged volume.

[0044] In overall appearance, the data processing and work station unit30 illustrated in FIG. 1 may be laid out in a conventional fashion, witha display section in which, for example, a previously acquired CT ordiagnostic image is displayed on one screen 32 while one or moreintraoperative images 33, such as a A/P and a lateral fluoroscopic view,are displayed on another screen. FIG. 2 schematically represents onesuch display. In its broad aspects the system may present an appearancecommon to many systems of the prior art, but, in a first aspect providesenhanced or corrected navigation guiding images, while in a secondaspect may provide CT or other reconstructed images in display 32 formeddirectly from the fluoroscopic views. In a third aspect the system mayprovide dynamic referencing between these reconstructed images and a setof preoperative 3D image data.

[0045] Typically, for tool positioning, one fluoroscope image in display33 may be taken with the beam disposed vertically to produce an A/Pfluoroscopic image projected against a horizontal plane, while anothermay be taken with beam projected horizontally to take a lateral viewprojected in a vertical plane. As schematically illustrated therein, theimage typically shows a plurality of differently shaded features, sothat a patient's vertebra, for example, may appear as an irregularthree-dimensional darkened region shadow-profiled in each of the views.The tool representation for a navigation system may consist of abrightly-colored dot representing tip position and a line or vectorshowing orientation of the body of the tool approaching its tip. In theexample shown in FIG. 2, in the horizontal plane, the probe projectedimage 40′ may extend directly over the imaged structure from the side inthe A/P or top view, while when viewed in the vertical plane theperspective clearly reveals that the tip has not reached that featurebut lies situated above it in space. In a preferred implementation ofthe multi-image display console of the invention, the display employsposition data from the tracking assembly to display the fluoroscope'scurrent angle of offset from the baseline AP and lateral views. Surgeonshave generally become accustomed to operating with such images, anddespite the fact that the fluoroscopic images are limited by beingprojection images rather than 3D images, their display of approximateposition and orientation, in conjunction with the diagnostic image onpanel 32 which may also have a tool point representation on it, enablesthe surgeon to navigate during the course of a procedure. In a preferredembodiment of the present invention, this display is further enhanced byemploying position data from the tracking assembly to display thefluoroscope's current angle of offset from the baseline AP and lateralfluoroscope views. This may be done as shown in FIG. 2A, by marking thefluoroscope's tracked angle or viewing axis with a marker on a circlebetween the twelve o'clock and three o'clock positions representing theAP and lateral view orientations.

[0046] The nature of the enhancement or correction is best understoodfrom a discussion of one simple embodiment of the present invention,wherein a tracking system tracks the surgical instrument 40, and thesystem projects a representation 40′ of the tool on each of the imagesdetected by the image detector 24. This representation, while appearingas a simple vector drawing of the tool, is displayed with its positionand orientation determined in the processor by applying a projectivetransform and an inverting image distortion transformation to the actualtool coordinates determined by the tracking elements. Thus, it isdisplayed in “fluoroscope image space”, rather than displaying a simpletool glyph, or correcting the image to fit the operating roomcoordinates of the tool.

[0047]FIG. 3 illustrates one embodiment 50 of a suitable marker array,calibration fixture or standard ST for the practice of the invention. Asillustrated in this prototype embodiment, the fixture may includeseveral sheets 52 of radiolucent material, each holding an array ofradiopaque point-like markers 54, such as stainless steel balls.(hereafter simply referred to as BBs). The BBs may be of different sizesin the different planes, or may be of the same size. Preferably, theyare of the same size, e.g., about one or two millimeters in diameter,and preferably the one or more plates holding them are rigidly affixedat or near to the face of the camera imaging assembly so as to allowaccurate calibration of the entire volume of interest while occupying asufficiently small space that the camera may be positioned closely tothe patient. The illustrated calibration fixture 50 includes areleaseable clamp assembly 51, with a camming clamp handle 51 a,configured to attach directly on or over the face of the cameraassembly.

[0048] As shown in the system diagram, FIG. 4, operation of the systemproceeds as follows.

[0049] Initially, as noted above, a tracking element is associated witheach of the tool, the patient and the fluoroscope. Each tracking elementis secured against movement with respect to the structure it istracking, but advantageously, all three of those structures are free tomove. Thus, the fluoroscope may move freely about the patient, and boththe patient and the tool may move within the operative field.Preferably, the tracking element associated with the fluoroscope ispositioned on a calibration fixture 50 which is itself rigidly affixedto the camera of the fluoroscope as described above. The calibrationfixture may be removably attached in a precise position, and thetracking element T2 may be held in a rigid oriented body affixed to thefixture 50. The tracking element T2 (FIG. 3) may, for example, be apoint-origin defining tracking element that identifies the spatialcoordinates and orientation of its housing, hence, with a rigidcoordinate transform, also specifies the position and orientationcoordinates of the object to which it is attached. Thus, the trackingelement T2 may with one measurement determine the positions of allmarkers in the calibration fixture, and the position and orientation ofthe fixture itself or the horizontal surface of the camera assembly.

[0050] The illustrated marker plates may each be manufactured by NCdrilling of an array of holes in an acrylic, e.g., Lexan, and/or otherpolymer plate, with the BBs pressed into the holes, so that all markercoordinates are exactly known. Alternatively, marker plates may bemanufactured by circuit board microlithography techniques, to providedesired patterns of radiopaque markers, for example as metallizationpatterns, on one or more thin radiolucent films or sheets. Applicantsalso contemplate that the calibration assembly, rather than employingseparate sheets bearing the markers, may be fabricated as a single block50 of a suitable radiolucent material, such as a structural foam polymerhaving a low density and high stiffness and strength. In that case, asshown in FIG. 3A, holes may be drilled to different depths and BBmarkers may be pressed in to defined depths Z₁, Z₂ . . . at specificlocations to create the desired space array of markers in a solid foamcalibration block. One suitable material of this type is a structuralfoam of the type used in aircraft wings for lightweight structuralrigidity. This material may also be employed in separate thinmarker-holding sheets. In any case the selected polymer or foam, and thenumber and size of the markers, are configured to remain directly in theimaging beam of the fluoroscope device and be imaged in each shot, whilethe position of the fixture is tracked. The fixture materials areselected to avoid introducing any significant level of x-ray absorptionor x-ray scattering by the plates, sheets or block, and the size andnumber of markers are similarly chosen to avoid excessive shadowing ofthe overall image, while maintaining a sufficiently dense image levelfor their detectability, so that both the imaging source radiation leveland the resulting image density scale remain comparable to currentlydesired operating levels. Preferably, the BBs are arranged in a patternat one or more levels, with a different pattern at each level. Further,when more than one array at different depths is used, the patterns arepositioned so that as the source/camera alignment changes, BBs of onepattern cast shadows substantially distinct from those of the otherpattern(s).

[0051] As noted above, in accordance with a principal aspect of thepresent invention, the array of markers is imaged in each fluoroscopeshot. As shown in FIG. 4, the image display system of the presentinvention operates by first identifying markers in the image. This isdone in an automated procedure, for example, by a pipeline of grey levelthresholding based on the x-ray absorption properties of the markers,followed by spatial clustering based on the shape and size of themarkers. In the preferred embodiment having two or more planar sheets,each sheet has markers arranged in a particular pattern. The pattern ofeach sheet will be enlarged in the image by a scale that varies with thecone divergence and the distance of the marker sheet along the axis fromthe optical center (or x-ray source) to the detection surface. Themarker images will also be shifted radially away from the beam centeraxis due to the beam divergence. In the preferred embodiment, thecalibration fixture is positioned close to the image detection surface,and the markers lie in arrays distributed in planes placed substantiallyperpendicular to the optical axis and offset from the detection surface.In general, not all markers will be located in the image due toshadowing of some of markers, or occlusion of the marker by anotherobject of similar x-ray absorption response. In a prototype embodimentof the marker identification image processor, the candidate markers inthe image are first identified using image processing and then matchedwith corresponding markers in the fixture.

[0052] One suitable protocol takes a candidate marker P_(i) in imagecoordinates, assumes it is, e.g., marker number Q_(j) of sheet one, andthen determines how many other candidate markers support this match,i.e., line up with the expected projections of the remaining markers ofone array, e.g., in the pattern of sheet one. The number of candidatesmatching the known template or pattern of sheet one is totaled, and istaken as the score of that marker. This process is repeated to scoreeach candidate marker in the image, and an identification scored above athreshold is taken as correct when it leads to the highest score forthat candidate, and does not conflict with the identification of anotherhigh-scoring candidate. Scoring of the match is done by using theobservation that the ratio of distances and angles between line segmentson the same plane are invariant under perspective projection. When thearray has only about fifty to one hundred markers, the processor mayproceed on a point-by-point basis, that is, an exhaustive matchingprocess may be used to determine the correspondence between points. Whena larger number of markers are desired, the marker detection processorpreferably employs an optimization algorithm such as the Powell,Fletcher or a simplex algorithm. One particularly useful patternmatching algorithm is that published by Chang et al in PatternRecognition, Volume 30, No. 2, pp. 311-320, 1997. That algorithm is bothfast and robust with respect to typically encountered fluoroscopicdistortions. As applied to calibration markers of the present invention,the Chang alignment/identification algorithm may be accelerated relyingupon the fact that the marker fixture itself has a known markergeometry. For example, the marker identification module may predict theexpected positions in the image, and search for matches within a definedsmall neighborhood. The image processor calibration module includes apre-compiled table, for example, stored in non-volatile memory,indicating the coordinates of each marker of the pattern, and preferablyincludes tables of separation for each pair, and/or included angle foreach triplet of markers, to implement fast identification.

[0053] As noted above, when the calibration plates are rigidly affixedto the camera, only a single tracking element T2 is needed to determinethe positions of all the markers, which differ only by a rigid transform(e.g. a translation plus a rotation) from those of the tracking element.Otherwise, if one or more of the arrays of markers is carried in aseparately-positioned sheet or fixture, each such unit may be tracked bya separate tracking element. In either case, the array of markerpositions are determined in each fluoroscopic image frame from thetracking element T2 and from the fixed relative position coordinatesstored in the marker table.

[0054] Continuing with a description of FIG. 4, in accordance with aprincipal aspect of the invention, the camera is next calibrated usingthe marker identification information of the previous steps. The imagingcarried out by the fluoroscope is modeled as a camera system in whichthe optical center is located at the x-ray source and the imaging planeis located a distance F (focal length) away from it inside the cameraassembly. The optical axis is the line through the x-ray source andperpendicular to the horizontal face of the camera. The intersection ofthe optical axis and the image plane is defined as the piercing point.Certain imaging or distortion characteristics may also be measured bythe array of marker images, which thus determines a correctiveperspective transformation. A suitable algorithm is that described byRoger Tsai in his article on 3-D camera calibration published in theIEEE Journal of Robotics and Automation, Volume RA-3, No. 4, August1987, pp. 323-344. This model determines radial distortion in additionto parameters using an algorithm that takes as input the matched markerand image locations, estimates of focal length and information about thenumber of rows and columns in the projection image. This algorithm isreadily implemented with one or more planes of markers in the fixture 50or 50′. When the fluoroscope is sufficiently rigid that focus does notvary, a single plane of markers may be used to define the cameraparameters.

[0055] By providing a pattern of markers in a plane, the shifts inposition of those markers in the image define a local transformationthat corrects for radial distortion of the image, while non-occludingmarkers in two planes, or at two different positions along the z-axisare sufficient to identify the focus and the optical or center axis ofthe beam. Other models relying, for example, on defining a distortionmorphing transformation from the array of marker images may also beapplied. A pattern of markers may comprise a rectangular lattice, e.g.,one marker every centimeter or half-centimeter in two orthogonaldirections, or may occupy a non-periodic but known set of closely-spacedpositions. The calibration fixture may be constructed such that markersfill a peripheral band around the imaged tissue, to provide markershadow images that lie outside the imaged area and do not obscure thetissue which is being imaged for display. Preferably, however, themarkers are located in the imaged field, so that the imaging camera anddistortion transforms they define closely fit and characterize thegeometric imaging occurring in that area. In the preferred embodiment,the image processor removes the marker shadow-images from thefluoroscope image frame before display on the console 30 (FIG. 1), andmay interpolate or otherwise correct image values in the surroundingimage.

[0056] Continuing with a description of the processing, the processor inone basic embodiment then integrates tracked tool position with thefluoroscope shot. That is, having tracked the position of tool 40 viatracking element T₁, relative to the marker array 50 and trackingelement T₂, and having modeled the camera focus, optical axis and imageplane relative to the position of the fixture 50, the system thensynthesizes a projection image of the tool as it dynamically tracksmovement of the tool, and displays that tool navigation image on thefluoro A/P and/or lateral view of screen 33 (FIG. 1).

[0057] To display the tool position on an uncorrected fluoroscope image,the processor obtains the position of the front and back tips of thetool. These are fixed offsets from the coordinates of the trackingelement T1 associated with the tool. The tracker may also determine toolorientation relative to the patient from position and orientationrelative to the tracking element T3 on the patient at the time of imagecapture. Tracked position coordinates are converted to be relative tothe fixed tracking element on the camera, or so that all coordinatesreference the image to which the camera model applies. In a basic toolnavigation embodiment, the camera calibration matrix is then applied tothe front and back tip position coordinates of the tool to convert themto fluoroscope image space coordinates. These end point coordinates areconverted to undistorted two-dimensional image coordinates (e.g.,perspective coordinates) using the calculated focal length of thecamera, which are then converted to distorted two-dimensional imagecoordinates using the lens distortion factor derived from the matrix ofmarker positions. Corresponding pixel locations in the two-dimensionalfluoroscope image are determined using the x-scale factor, thecalculated origin of the image plane and scaling based on the number ofpixels per millimeter in the camera image sensor and display. Thedetermined position is then integrated with the video display on thefluoroscope to show a graphical representation of the tool with itsfront tip location in image coordinates. Preferably, the tool isdisplayed as an instrument vector, a two-dimensional line on thefluoroscopic image with a red dot representing its tip. Thereafter,during an ongoing procedure, the tracking assembly may track toolmovement relative to the patient, and a processor controls the trackingand determines from the position of the tool when it is necessary toredraw the integrated display using the above-described image distortiontransformations to correctly situate the displayed tool in a position ona new image.

[0058] As described above, the process of camera calibration is aprocess of applying actual coordinates as determined by the trackingsystem and marker positions, and image coordinates as seen in thefluoroscopic marker images, to model a camera for the image. In general,applicant's provision of an array of marker points having knowncoordinates in each of several planes, together with trackingcoordinates corresponding to the absolute position of those planes andmodeling of the camera image plane with respect to these trackedpositions obviates the need for lengthy initialization or correlationsteps, and allows an image processor to simply identify the markerimages and their positions in the image, model the camera to definefocus, image plane and piercing point, and to effect image correctionswith a few automated tracking measurements and transformations. Thefixture is preferably fixed close to the front surface of the imagedetector assembly, so the calibration fits the detected image closely.

[0059] As noted above, the marker positions allow a simple computationof effective parameters to fully characterize the camera. This allowsone to scale and correct positions of the image (for example a tool)when their coordinates are tracked or otherwise unknown.

[0060] In accordance with a preferred method of operation of the presentdevice, the fluoroscope is operated to take a large number of fluoroimages, with fixture tracking and camera modeling as described above,and a 3D CT image data set is reconstructed from the acquired data. Ingeneral, this data set can be acquired such that it is dimensionallyaccurate and useful for close surgical guidance, although parameterssuch as x-ray absorbance, corresponding, for example to bone or tissuedensity, will be of lesser accuracy than those obtainable from a CTscanner, and should not be relied upon. The fluoroscopic CT images soformed may be further correlated with preoperative MRI, PET or CT imagesto define a direct image coordinate transformation, using establishedtechniques such as MI (mutual information) registration, edge or contourmatching, or the like, between the fluoroscopic 3D data set of thepresent invention and the existing preoperative 3D image set.

[0061] Operation for forming a volume image data set for CTreconstruction proceeds as follows. First, the fluoroscope is operatedto obtain a dense set of fluoroscope images, for example, by rotatingthe fluoroscope approximately in a plane about the patient through 180°plus the angle of divergence of the cone beam, taking a shot everydegree or less, so as to image a particular three-dimensional tissuevolume of the patient in a large number of images. As each frame isacquired, pose information, given for example by the position andorientation measurement of the tracking element T2, is stored, and themarker detection/calibration module operates on each shot so that acorrection factor and a perspective projection matrix is determined foreach image, as described above, to model the camera focus, image planeand optical axis for that shot. A coordinate system for the tissuevolume for which reconstruction is desired is then computed, and theprocessor then applies filtered back projection or other reconstructionprocessing (such as lumigraphs or lambda-CT), with indexing provided bythe relative disposition of each pose, to reconstruct athree-dimensional volume data image set in the intra-operativecoordinate system for a region of tissue around the origin of thereconstruction coordinate system. This 3-D image data set referenced totracker coordinates readily allows CT reconstruction of desired planeswithin the image set, referenced to patient or tool position.

[0062] In order to integrate the tracking system with the fluoroscopicimages, it is necessary to establish a coordinate system for thethree-dimensional reconstructed volume. This entails defining the originand the coordinate axes for that volume. Once such a coordinate systemis defined in relation to all fluoro images, one can compute the backprojection at voxels in a region referenced to the origin, in planesthat are perpendicular to one of the coordinate axes. In the case of aspinal scan, for example, the desired CT planes will be planesperpendicular to an axis that approximates the long axis of the body.Such a spinal data set is especially useful, since this view cannot bedirectly imaged by a fluoroscope, and it is a view that is critical forvisually assessing alignment of pedicle screws. Applicant establishesthis common coordinate system in a way that minimizes risk of: (a)backprojecting voxels where insufficient data exists in the projectionsor (b) being unable to define the relationship between the naturalcoordinate system of the patient and that of the reconstruction.

[0063] In the discussion that follows, it will be assumed that the user,e.g., the surgeon, or radiologist takes the fluoroscopic images suchthat the region of interest stays visible, preferably centered, in allthe fluoroscopic images and that each arc traced by the C-arm isapproximately planar. These requirements may be met in practice by usersof C-arm fluoroscopes, since surgeons have extensive practice inacquiring fluoroscopic images by tracing planar motion trajectories inwhich the relevant anatomy is centered in both AP and lateral views.Such centering is easiest to achieve, or most accurately attained whenusing a substantially isocentric C-arm such as that made by the Siemenscorporation. However, in calibrating the camera for each image,applicants are able to automatically determine a reconstructioncoordinate system for an arbitrary sequence of images. In this regard,the camera tracking data may be used to fit a center. This is consideredan advance over systems that require a coordinate system to be specifiedmanually.

[0064] It will be appreciated that in the above-described system, thetracking elements automatically detect coordinates of the marker array,tool and patient at the time each image is taken. Detection of thecalibration fixture position allows camera modeling to provide theposition of the optical center (F), optical axis and image plane, intracker coordinates for each shot as described above. In accordance withthis further aspect of the invention, the combination of trackedposition and modeled camera information is used to define a coordinatesystem for the reconstruction, which is preferably computed byperforming statistical and computational geometry analysis on the poseinformation recorded and derived for each of the fluoroscopic imageframes.

[0065] A few definitions will clarify the underlying procedure, employedin the prototype embodiment and automated in software. The “projectionplane” is the plane on which the image is formed through the operationof perspective projection. The “optical center” or the “center ofprojection”, C, is located at a distance F, the focal length of theoptical system, from the projection plane. In the case of a fluoroscope,this is the actual location of the x-ray source; the source ispositioned at the optical center of the imaging system. The projectionof a given point M in the world is computed as the intersection of theray connecting M and the optical center C with the projection plane. The“optical axis” of a fluoroscopic imaging system is the line that passesthrough its optical center (the x-ray source) and is normal to theprojection plane. The point at which the optical axis intersects theprojection plane is known as the “principal point” or the “piercingpoint”. A textbook such as “Three-Dimensional Computer Vision” byOlivier Faugeras, MIT Press, may be consulted for further background orillustration of basic concepts used here.

[0066] Applicant's approach to the problem of computing a coordinateorigin for reconstruction assures that in this set of data, the originof the 3D coordinate system lies at a point that is the center of theregion that the surgeon is most interested in visualizing. That point isidentified in a prototype system by computing a point that is closest tobeing centered in all of the acquired fluoroscopic images, and thentaking that point as the origin of a coordinate system in which thereconstruction is performed. FIG. 5 sets forth the steps of thisprocessing.

[0067] It will be recalled that the camera calibration described abovemodels the camera for each shot. Each configuration of the C-arm definesa coordinate system in which the origin, (0,0,0) is defined by thelocation of the x-ray source. The principal point is located at (0,0,F)where F is the focal length. That is, the optical axis, or axis of theimaging beam, is aligned along the third axis. Such a situation isschematically illustrated in FIG. 7 for the two fluoroscope positionsshown in FIG. 1A. If all the fluoroscope configurations are taken in thecontext of a common world-coordinate system, each of theseconfigurations defines a unique optical axis. Ideally, the point inthree-space where all these optical axes intersect would be visible andcentered in all the projection images. Based on the assumption that thefluoroscopic images are acquired by approximately centering the regionof interest, applicant defines a projection center of the imaged tissuevolume from the ensemble of camera models, and uses this intersectionpoint as the origin for a three-dimensional reconstruction. This is doneby applying a coordinate determination module, which identifies theoptical axis intersection point as the intersection of, or best fit to,the N² pairs of optical axes of the modeled cameras for the N poses. Inpractice, two facts should are addressed in computing the center ofprojection: (a) the optical axes of any two fluoroscope shots areusually somewhat skew, lying in separate, but substantially parallelplanes, and do not really intersect in space, and (b) the two“intersection” points determined by two different pairs of axes also donot generally coincide exactly.

[0068] In order to address these problems, for situation (a), theprocessor incorporates a software condition check for skewness of lines.If the optical axes are skew, the processor defines the intersectionpoint as a computed point that is halfway between the two lines. Inorder to address the situation (b), the processor takes the meancoordinates of the N² skew-intersection points determined in the firststep as its common center of projection. Thus the cluster of pointsdefined by the N² pairs of axes determines a single point. This point isdefined as the origin of the tissue region for which reconstruction isundertaken.

[0069] It is also necessary to determine a set of coordinate axes forthe volume data set. Preferably, the axial planes of the reconstructionare to be parallel to the plane of motion of the x-ray source.Applicant's presently preferred processing module computes the plane ofmotion of the x-ray source by fitting a least-squares solution to theposes of the x-ray source. Any two non-collinear vectors in this planedefine a basis for this plane and serve as two of the axes for thecoordinate system. The module also computes a normal to this plane toserve as the third coordinate axis. The coordinate axis computation maybe done by using eigen-analysis of the covariance matrix of thecoordinates of the optical centers (x-ray source locations) and theprincipal points in the world-coordinate system. These eigenvectors arethen ordered in order of decreasing eigenvalue. The first twoeigenvectors provide a basis for the axial plane of interest, and thethird eigenvector provides the normal to this plane. This procedure thusprovides all three coordinate axes for the three-dimensionalreconstruction. This determination is fully automated, and requires onlythe tracker data and camera models determined by the processor when eachshot is taken. Further background and details of implementation forapplying the eigenanalysis technique to define coordinate axes may befound in reference texts, such as the 1984 textbook “PatternRecognition” by J. Therrien.

[0070] Having determined a coordinate system for the reconstruction, theprocessor then filters and back-projects the image data to form a volumeimage data set, from which CT planes may be reconstructed or retrievedin a conventional manner. The back projection step may utilize fast orimproved processes, such as the fast Feldkamp algorithm or othervariant, or may be replaced by other suitable volume data reconstructiontechnique, such as the local or Lambda tomography method described by A.Louis and P. Maass in IEEE Transac. Med. Imag. 764-769, (1993) andpapers cited therein.

[0071] Thus, a simple set of automated tracking elements combined withimage processing operative on a fixed or tracked marker array providesaccurate tool tracking fluoroscope images, or a set of geometricallyaccurate reconstructed or CT images from the shadowgraphic images of aC-arm or intraoperative fluoroscope. The nature of the multi-pointmarker-defined camera image model allows the processor to quicklyregister, reference to a common coordinate system and back project orotherwise reconstruct accurate volume image data, and the fastdetermination of a camera parameter model for each shot proceeds quicklyand allows accurate tool display for intraoperative tool navigation anddynamic tracking, without requiring rigid frames or robotic assembliesthat can obstruct surgery, and without the necessity of matching to anMRI or PET database to achieve precision. Furthermore in the preferredembodiment, the models, transformations and fitting to a coordinatesystem proceed from the tracker position measurements of the markerfixture relative to the patient or tool, rather than from an extrinsicfixed frame, reducing potential sources of cumulative errors andsimplifying the task of registering and transforming to commoncoordinates. Applicant is therefore able to precisely track and displaythe tool in real time, and to produce accurate fluoro-CT images using aC-arm fluoroscope.

[0072] It will be understood that the description above relies upontracking measurements made by tracking elements each fixed with respectto one of a few movable objects. As applied to the patient or tool,these tracking elements may be affixed by belts, frames, supports,clips, handles or other securing or orienting structures known in theart. In general, applicant's preferred tracking element is a magneticfield tracking element, which may be oriented and affixed in a rigidhousing that allows it to secure to the structure to be tracked. Actualimplementation of the system may involve a preliminary calibrationprocedure wherein the actual dimension, offset or relative position ofthe tool tip, the marker array or the like, with respect to the tool orarray tracking element is permanently stored in a chip or non-volatilememory so that minimal or no set-up initialization is required during animaging session. Similarly, when employing such magnetic trackingelements, a table of field or position corrections may be initiallycompiled for the tracking element mounted on the fluoroscope to assurethat the tracking achieve a high level of accuracy over a broad fieldextending quite close to the image detector and C-arm structures.Additional reference sensor-type tracking elements or standards may alsobe provided as described in the aforesaid '980 patent, if desired toenhance the range, resolution or accuracy of the tracking system.

[0073] The calibration fixture has been described above as beingpreferably affixed to the image detector portion of the fluoroscope,where, illustratively one or several precision arrays of markers locatedalong the imaging axis provide necessary data in the image itself tocharacterize the camera each time an image is taken. This location, withthe markers in a single fixture, provides a high level of accuracy indetermining the desired camera parameters, and enables tracking toproceed without obstructing the surgical field.

[0074] To the extent that the constraint of positioning the calibrationfixture between the target tissue and the detector may limit flexibilityin positioning the image detector near the patient, this may beaddressed in other embodiments by having all or a portion of the markerarray assembly implemented with markers located on or in a radiographicsupport table (75, FIG. 6) or other structure on which the patient orthe imaged tissue portion is supported. In this case, the table orsupport itself, which is radiolucent, may have a thickness and structurethat permits markers to be embedded at different depths. For example, itmay be formed of a structural foam material as described above in regardto the marker fixture of FIG. 3A. Alternatively, the markers may beincluded in one or more sheets that fit within the x-ray sheet film trayof a conventional radiography table, or such marker sheets may belaminated to the bottom and/or top surfaces of the table. When affixedto the table or inserted in a registered or fixed fashion, the trackingelement T₂ may then be attached anywhere on the rigid structure of thetable itself, with suitable offsets stored in a fixed memory element ofthe system. In embodiments utilizing such markers in the table, thetotal angular range of the poses in which useful marker images willappear in the fluoroscope images may be restricted to somewhat under180°. Furthermore, the image plane will generally not be parallel to themarker arrays, so a different set of computations is utilized by theprocessor to characterize the camera position and geometry. However,these computations involve straightforward camera modeling, and may beaccelerated by also tracking the image detector with an additionalelement T₂′.

[0075] The calibration fixtures of the invention as well as theembodiments having markers on or in the table may be implemented withone or more separately-made sheet structures. FIG. 6 shows elements ofone such embodiment wherein a marker array 50″ is formed as a pattern ofmetallized dots 56, which may be formed lithographically on aprinted-circuit type sheet. As indicated schematically in this Figure,the sheet may also bear one or more lithographically-formed conductiveloops 58, configured as a field generating or field sensing loop, fordefining one or more elements of a magnetic tracking assembly. Three ormore such patterned loops may be formed to constitute a basicelectromagnetic generator or sensor that advantageously is preciselypre-aligned with respect to the coordinates of the markers 56 by virtueof its having been manufactured using a pattern lithography mask. Themagnetic circuit loops may define magnetic multipoles for establishingor sensing position-tracking electromagnetic fields, or may, forexample, include one or more coils of a system of Helmholtz coils forestablishing a gradient field in the region where tracking is to occur.These may operate in conjunction with other coils disposed elsewhere fordefining the tracking field, The implementation of magnetic tracking andradiographic marker elements on a sheet also allows plural sheets to bepositioned and tracked separately for effecting the imaged basedprocessing of the present invention.

[0076] In addition to the above described structure and operation of theinvention, applicant contemplates system embodiments wherein a fluoro-CTdata set is constructed as described above, and the fluoro-3D data setis then registered or correlated to an existing MRI, CT or PET 3D dataset to form a fused set of images. These are then displayed on thesystem console 30 (FIG. 1) to provide enhanced patient informationduring surgery. Advantageously, the coordinates of the fluoro-CT imagesare known from the coordinates used in the reconstruction processing,while the correlation of the two different 3D image sets may proceedwithout reference to patient or other tracking coordinates, using anyconventional 3D registration or correlation technique. This providesfast and effective fused image sets for surgical guidance or diagnosticevaluation.

[0077] Indeed, the system need not produce detailed fluoro-CT images, orneed not display those images. Instead, the fluoro-CT images, or alesser quality set of fluoro-CT images constructed from a faster(smaller) scan sequence of fluoro images, defined in trackercoordinates, may be produced and simply registered to a preoperative 3Ddata set in order to bring that preoperative image data set into thetracker coordinate system. In that case, the system applies thisregistration, and proceeds thereafter by simply tracking the patient andthe tool, and displaying the appropriate preoperative images for eachtracked location as shown in FIG. 8. Thus, in accordance with thisaspect of the invention, the system provides an automated registrationsystem for the intraoperative display of preoperative MRI, PET or CTimages, without requiring placement or imaging of fiducials, withoutrequiring the surgeon to initialize or set up a plurality of referencepoints, without requiring the surgeon to cut down to or expose a fixedskeletal registration feature, and without requiring immobilization ofthe patient in a frame or support. Instead, the intermediate fluoro-CTimages are produced as part of an automated modeling and coordinatizingprocess, and both the production and the registration of these images tothe preoperative data set may proceed entirely automated in software,for example, registering by mutual information (MI), feature correlationor similar process.

[0078] The invention being thus disclosed and illustrative embodimentsdepicted herein, further variations and modifications of the invention,will occur to those skilled in the art, and all such variations andmodifications are considered to be within the scope of the invention, asdefined by the claims appended hereto and equivalents thereof. Each ofthe patents and publications identified above is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. A system for surgical imaging and display of thetype indicating tool position together with an image of a patientundergoing surgery, and including a display and an image processor fordisplaying such image in coordination with an image of the tool forvisual navigation of the tool during the surgical procedure, the systembeing configured for use with a fluoroscope that includes an x-raysource and an imaging system, and such that at least one image in thedisplay is derived from the fluoroscope at the time of surgery, whereinthe system comprises: a fixture including a three dimensional array ofmarkers and adapted to be rigidly affixed to a fluoroscope forautomatically imaging the array of markers as a fluoroscope image isacquired with the fluoroscope; a tracking assembly including a firsttracking element associated with the tool and a second tracking elementassociated with the fixture, the tracking assembly being effective toautomatically determine relative position data of the marker array andthe tool; a marker determination unit operative on the fluoroscope imageto identify a subset of said markers in the image; a processor operativewith said identification to calibrate the system; and a correction unitresponsive to the system calibration and to said position data todetermine an image transformation such that a representation of the toolis correctly positioned and oriented in the fluoroscope image.
 2. Thesurgical imaging and display system of claim 1, wherein said fixtureincludes arrays of markers arranged in two different planes along animaging direction in substantially non-shadowing patterns, and saidcorrection unit applies the marker position data to determine camerafocus and center to register the representation of the tool in thefluoroscope image.
 3. The surgical imaging and display system of claim1, wherein said correction unit determines an image distortion inversetransform operative to convert tracking assembly spatial coordinates ofthe tool relative to the camera, to fluoroscope image coordinates. 4.The surgical imaging and display system of claim 1, wherein saidprocessor determines a camera projection geometry for each of aplurality of fluoroscope images and applies said camera projectiongeometry to identify a tissue coordinate system for a common region ofimaged tissue.
 5. The surgical imaging and display system of claim 4,wherein said processor transforms image data of said plurality offluoroscope images to a 3D fluoro-CT data set representative of saidcommon image region.
 6. The surgical imaging and display system of claim1, wherein said processor determines a camera model that includes radialdistortion of said fluoroscope image.
 7. The surgical imaging anddisplay system of claim 1, further comprising a table of calibrationdata representing static camera corrections that vary with cameraposition or orientation.
 8. The surgical imaging and display system ofclaim 1, wherein the fixture is releasably attachable to a front surfaceof an image detector assembly of the fluoroscope camera.
 9. The surgicalimaging and display system of claim 1, wherein the three dimensionalarray of markers includes a first array of markers disposed in a firstplane, and a second array of markers disposed in a second plane parallelto the first plane, the planes having a defined separation.
 10. Thesurgical imaging and display system of claim 1, wherein the threedimensional array of markers includes a first array of markers disposedin a first plane, and a second array of markers disposed in a secondplane, wherein one of said planes is affixed to the camera, and atracking element is secured against movement with respect to the otherof said planes.
 11. The surgical imaging and display system of claim 1,wherein said fixture comprises a radiolucent and substantiallynon-scattering body having a plurality of radio-opaque features atdifferent depths arranged to constitute patterns of markers imaged bythe image detector.
 12. The surgical imaging and display system of claim1, wherein one of said tracking elements is an electromagnetic trackingelement that includes a lithographically formed sheet assemblycomprising marker elements.
 13. A system for use with a fluoroscope thatincludes a source and an image detector, wherein the system comprises: afixture including a three dimensional array of point-likefluoroscopically opaque markers disposed over a region in a rigid body,said body being configured for disposition between the source and camerasuch that the array of markers is automatically imaged as an image isacquired by the fluoroscope; a tracking assembly including at least afirst tracking element associated with the fixture and configured forautomatically determining position data of the fixture with respect tothe tool; a marker determination unit operative on the fluoroscope imageto identify a subset of said markers in the image and operative with atable of predefined marker positions to determine camera imagingparameters for the fluoroscope image; and a correction unit responsiveto said camera imaging parameters and to said position data to form acorrected tool navigation fluoroscopic image.
 14. A system for displayof tissue images and probe images during a surgical procedure on apatient, such system comprising a fixture holding a plurality ofradiopaque markers disposed in predefined patterns, said fixture beingpositionable between a source and an imaging side of a fluoroscope; astored table of position of said markers; an image processor operativeon a fluoroscopic image to identify imaged markers in the fluoroscopicimage and operative with said stored table to model at least cameraimage plane piercing point and focus data for said image; a trackerincluding a first tracking element securable against movement withrespect to said fixture and at least one further tracking elementsecurable against movement with respect to an object of interest such asa tool or a patient, said tracker determining coordinates of saidfixture and said object of interest, wherein said image processorapplies said coordinates together with image positions of the identifiedimaged markers to model the camera data; and an image display thatapplies said camera data to display a corrected fluoroscope image fortool guided surgery.
 15. The tissue image display system of claim 14,wherein said processor and image display remove images of the markersfrom the fluoroscope image to display an interpolated unobstructedfluoroscope image.
 16. The tissue image display system of claim 14,wherein the stored table is stored in nonvolatile memory.
 17. The tissueimage display system of claim 16, wherein the tracker assembly includesa table of tracker position correction data for a given fluoroscope. 18.The tissue image display system of claim 14, wherein the trackerassembly includes a field generating tracking element and a plurality offield sensing tracking elements, respective ones of the trackingelements being securable against movement with respect to the patient, atool and the fluoroscope imaging detector such that the system modelssaid camera data and corrects images of the patient and toolirrespective of movement of the patient, the tool and the fluoroscope.19. A system for use with a fluoroscope camera device for intraoperativetracking during a procedure on a patient, such system being of the typeincluding a display for displaying one or more fluoroscope imagestogether with an indication of probe or tissue status or position, andwherein the system comprises: a fixed multi-dimensional array offluoroscopically-imageable markers affixed on or proximate to an imagingassembly of the fluoroscope such that the markers are imaged in eachview; a tracking assembly including at least a first tracking elementreferenced to a patient and a second tracking element referenced to saidfixed array; a marker localization and identification unit operative onimage data to identify in real time the position of markers in eachfluoroscopic image and to replace the marker images with smooth imagevalues to thereby create an unobstructed real time patient fluoroscopicview of patient tissue; and a processor operative on detected imagemarkers for integrating said view with patient position, tool positionor preexisting image data to depict a trajectory of the surgical probein said patient tissue intraoperatively in real time.
 20. The system ofclaim 19, wherein said tracking assembly tracks position of a patientwith respect to the marker array, and an image registration unitoperates with data from the tracking assembly and the tracked markers toregister the dimensionally corrected image to the patient.
 21. Thesystem of claim 20, wherein said processor determines a projectiongeometry camera calibration for each of a plurality of fluoroscopeimages, and further determines a common coordinate system about a regionof tissue imaged by the fluoroscope, said processor further applying atransformation of image data of said plurality of images to define afluoro-CT image data set representative of said region of tissue. 22.Apparatus for surgical imaging and probe guidance, such apparatuscomprising: a fluoroscope having a radiation emitting side and animaging side, said fluoroscope being moveable to provide two-dimensionalprojection images through the body of a patient placed in its imagingfield a tracking system effective to simultaneously determine relativeposition of a patient, the fluoroscope and a surgical probe acalibration fixture securing imageable registration markers positionedsuch that a plurality of said markers are identifiably imaged in everyimage taken by the fluoroscope, said calibration fixture being affixedto the fluoroscope so that the fixture is tracked by the tracking systemat least some of said markers lying at different coordinate positionsalong an imaging axis and being characterized by defined positions withrespect to each other such that angle and spacing of images of saidmarkers are effective to characterize camera parameters for relating avolume region of the patient imaged in each fluoroscopic image to camerafocus and optical center, and an image processor that operates on imagesof said markers to model fluoroscope image projection relative totracked coordinates for each fluoroscope view such that imaged toolposition in successive fluoroscope images taken from differentorientations is accurately registered in successive images to trackactual movement of the probe with respect to the patient.
 23. Thesurgical imaging system of claim 22, wherein said image processorcorrects image coordinates to projected tracked coordinate of a probe,to provide accurate tracking and display of a probe by measurement ofprobe position and change of display coordinates so as to display probeposition on a display image without re-imaging of the tissue or probe.24. A calibration fixture for use with a fluoroscope, such fixturecomprising a radiolucent sheet or body having a major surface and athickness, said fixture including an array of closely spaced radioopaquemarkers distributed over a region at or below said major surface atdefined coordinate positions and being configured to secure in a fixedposition in the imaging beam of a flouroscope such that a plurality ofthe markers are imaged in each pose and images of the markers areeffective to define a camera model for the pose.
 25. The calibrationfixture of claim 24, wherein the sheet or body is formed of a rigid foamconfigured to have low x-ray absorption and scattering characteristics.26. The calibration fixture of claim 25, wherein the array includesfirst and second sub-arrays of markers disposed at first and secondoffsets from said major surface.
 27. The calibration fixture of claim24, mounted in a radiographic patient support so as to be imagedtogether with the patient.
 28. The calibration fixture of claim 24,adapted to be releasably affixed to a fluoroscope image detector toimage the markers in each image formed by the fluoroscope.
 29. Thecalibration fixture of claim 24, wherein the marker array islithographically formed together with a co-registered magnetic element.30. A calibration fixture comprising a sheet body having (i) an array ofradioopaque markers distributed over a major region of the sheet, and(ii) a magnetic tracking element attached to the sheet in definedrelation to the array of markers.
 31. The calibration fixture of claim30, wherein the magnetic tracking element is at least partly formed onthe sheet in registration with a pattern of said markers.